<html>
<body>

<p>This module enables search result grouping with Lucene, where hits
with the same value in the specified single-valued group field are
grouped together.  For example, if you group by the <code>author</code>
field, then all documents with the same value in the <code>author</code>
field fall into a single group.</p>

<p>Grouping requires a number of inputs:</p>

  <ul>
    <li> <code>groupField</code>: this is the field used for grouping.
      For example, if you use the <code>author</code> field then each
      group has all books by the same author.  Documents that don't
      have this field are grouped under a single group with
      a <code>null</code> group value.

    <li> <code>groupSort</code>: how the groups are sorted.  For sorting
      purposes, each group is "represented" by the highest-sorted
      document according to the <code>groupSort</code> within it.  For
      example, if you specify "price" (ascending) then the first group
      is the one with the lowest price book within it.  Or if you
      specify relevance group sort, then the first group is the one
      containing the highest scoring book.

    <li> <code>topNGroups</code>: how many top groups to keep.  For
      example, 10 means the top 10 groups are computed.

    <li> <code>groupOffset</code>: which "slice" of top groups you want to
      retrieve.  For example, 3 means you'll get 7 groups back
      (assuming <code>topNGroups</code> is 10).  This is useful for
      paging, where you might show 5 groups per page.

    <li> <code>withinGroupSort</code>: how the documents within each group
      are sorted.  This can be different from the group sort.

    <li> <code>maxDocsPerGroup</code>: how many top documents within each
      group to keep.

    <li> <code>withinGroupOffset</code>: which "slice" of top
      documents you want to retrieve from each group.

  </ul>

<p>The implementation is two-pass: the first pass ({@link
  org.apache.lucene.search.grouping.TermFirstPassGroupingCollector})
  gathers the top groups, and the second pass ({@link
  org.apache.lucene.search.grouping.TermSecondPassGroupingCollector})
  gathers documents within those groups.  If the search is costly to
  run you may want to use the {@link
  org.apache.lucene.search.CachingCollector} class, which
  caches hits and can (quickly) replay them for the second pass.  This
  way you only run the query once, but you pay a RAM cost to (briefly)
  hold all hits.  Results are returned as a {@link
  org.apache.lucene.search.grouping.TopGroups} instance.</p>

<p>
  This module abstracts away what defines group and how it is collected. All grouping collectors
  are abstract and have currently term based implementations. One can implement
  collectors that for example group on multiple fields.
</p>

<p>Known limitations:</p>
<ul>
  <li> For the two-pass grouping collector, the group field must be a
    single-valued indexed field.
    {@link org.apache.lucene.search.FieldCache} is used to load the {@link org.apache.lucene.search.FieldCache.StringIndex} for this field.
  <li> Although Solr support grouping by function and this module has abstraction of what a group is, there are currently only
    implementations for grouping based on terms.
  <li> Sharding is not directly supported, though is not too
    difficult, if you can merge the top groups and top documents per
    group yourself.
</ul>

<p>Typical usage for the generic two-pass collector looks like this
  (using the {@link org.apache.lucene.search.CachingCollector}):</p>

<pre class="prettyprint">
  TermFirstPassGroupingCollector c1 = new TermFirstPassGroupingCollector("author", groupSort, groupOffset+topNGroups);

  boolean cacheScores = true;
  double maxCacheRAMMB = 4.0;
  CachingCollector cachedCollector = CachingCollector.create(c1, cacheScores, maxCacheRAMMB);
  s.search(new TermQuery(new Term("content", searchTerm)), cachedCollector);

  Collection&lt;SearchGroup&lt;BytesRef&gt;&gt; topGroups = c1.getTopGroups(groupOffset, fillFields);

  if (topGroups == null) {
    // No groups matched
    return;
  }

  boolean getScores = true;
  boolean getMaxScores = true;
  boolean fillFields = true;
  TermSecondPassGroupingCollector c2 = new TermSecondPassGroupingCollector("author", topGroups, groupSort, docSort, docOffset+docsPerGroup, getScores, getMaxScores, fillFields);

  //Optionally compute total group count
  TermAllGroupsCollector allGroupsCollector = null;
  if (requiredTotalGroupCount) {
    allGroupsCollector = new TermAllGroupsCollector("author");
    c2 = MultiCollector.wrap(c2, allGroupsCollector);
  }

  if (cachedCollector.isCached()) {
    // Cache fit within maxCacheRAMMB, so we can replay it:
    cachedCollector.replay(c2);
  } else {
    // Cache was too large; must re-execute query:
    s.search(new TermQuery(new Term("content", searchTerm)), c2);
  }

  TopGroups&lt;BytesRef&gt; groupsResult = c2.getTopGroups(docOffset);
  if (requiredTotalGroupCount) {
    groupsResult = new TopGroups&lt;BytesRef&gt;(groupsResult, allGroupsCollector.getGroupCount());
  }

  // Render groupsResult...
</pre>

<p>To use the single-pass <code>BlockGroupingCollector</code>,
   first, at indexing time, you must ensure all docs in each group
   are added as a block, and you have some way to find the last
   document of each group.  One simple way to do this is to add a
   marker binary field:</p>

<pre class="prettyprint">
  // Create Documents from your source:
  List&lt;Document&gt; oneGroup = ...;
  
  Field groupEndField = new Field("groupEnd", "x", Field.Store.NO, Field.Index.NOT_ANALYZED);
  groupEndField.setOmitTermFreqAndPositions(true);
  groupEndField.setOmitNorms(true);
  oneGroup.get(oneGroup.size()-1).add(groupEndField);

  // You can also use writer.updateDocuments(); just be sure you
  // replace an entire previous doc block with this new one.  For
  // example, each group could have a "groupID" field, with the same
  // value for all docs in this group:
  writer.addDocuments(oneGroup);
</pre>

Then, at search time, do this up front:

<pre class="prettyprint">
  // Set this once in your app & save away for reusing across all queries:
  Filter groupEndDocs = new CachingWrapperFilter(new QueryWrapperFilter(new TermQuery(new Term("groupEnd", "x"))));
</pre>

Finally, do this per search:

<pre class="prettyprint">
  // Per search:
  BlockGroupingCollector c = new BlockGroupingCollector(groupSort, groupOffset+topNGroups, needsScores, groupEndDocs);
  s.search(new TermQuery(new Term("content", searchTerm)), c);
  TopGroups groupsResult = c.getTopGroups(withinGroupSort, groupOffset, docOffset, docOffset+docsPerGroup, fillFields);

  // Render groupsResult...
</pre>

Note that the <code>groupValue</code> of each <code>GroupDocs</code>
will be <code>null</code>, so if you need to present this value you'll
have to separately retrieve it (for example using stored
fields, <code>FieldCache</code>, etc.).

<p>Another collector is the <code>TermAllGroupHeadsCollector</code> that can be used to retrieve all most relevant
   documents per group. Also known as group heads. This can be useful in situations when one wants to compute grouping
   based facets / statistics on the complete query result. The collector can be executed during the first or second
   phase.</p>

<pre class="prettyprint">
  AbstractAllGroupHeadsCollector c = TermAllGroupHeadsCollector.create(groupField, sortWithinGroup);
  s.search(new TermQuery(new Term("content", searchTerm)), c);
  // Return all group heads as int array
  int[] groupHeadsArray = c.retrieveGroupHeads()
  // Return all group heads as FixedBitSet.
  int maxDoc = s.maxDoc();
  FixedBitSet groupHeadsBitSet = c.retrieveGroupHeads(maxDoc)
</pre>

</body>
</html>
